{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "21599c07",
   "metadata": {},
   "source": [
    "# Fluency Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use the Fluency evaluator to assess the linguistic quality of AI-generated responses. The evaluator measures how well generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage.\n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7add5ec1",
   "metadata": {},
   "source": [
    "The Fluency evaluator assesses the extent to which generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage, resulting in linguistically correct responses.\n",
    "\n",
    "Fluency scores range from 1 to 5:\n",
    "\n",
    "<pre>\n",
    "Score 1: Very Poor - The response is incomprehensible with severe grammatical errors and improper vocabulary.\n",
    "Score 2: Poor - The response has frequent grammatical errors and awkward phrasing that hinder understanding.\n",
    "Score 3: Fair - The response is understandable but contains noticeable grammatical errors or awkward expressions.\n",
    "Score 4: Good - The response is mostly fluent with minor grammatical issues that don't significantly impact readability.\n",
    "Score 5: Excellent - The response is perfectly fluent with proper grammar, syntax, and vocabulary usage.\n",
    "</pre>\n",
    "\n",
    "The evaluation requires one of the following input patterns:\n",
    "\n",
    "**Pattern 1: Single Response Evaluation**\n",
    "- Response: The text response to be evaluated for fluency. (string)\n",
    "\n",
    "**Pattern 2: Conversation Evaluation**\n",
    "- Conversation: A multi-turn conversation where each turn is evaluated for fluency. (List[dict])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2abd4d05",
   "metadata": {},
   "source": [
    "### Initialize Fluency Evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfd097f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import FluencyEvaluator, AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "fluency_evaluator = FluencyEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4793759a",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46dc8414",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c2df39a",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = \"The weather in Seattle is currently partly cloudy with a temperature of 15°C. The forecast indicates that conditions will remain stable throughout the day, with a gentle breeze from the northwest.\"\n",
    "\n",
    "result = fluency_evaluator(response=response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59709c65",
   "metadata": {},
   "source": [
    "#### Example of Poor Fluency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9c624ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Poor fluency example\n",
    "response = \"Email draft attach is. You review and giving feedback must. Important very for project success it being.\"\n",
    "\n",
    "result = fluency_evaluator(response=response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b26dd4bb",
   "metadata": {},
   "source": [
    "#### Conversation as Dict with Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ccaff56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi-turn conversation\n",
    "conversation = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Can you help me understand the weather forecast for this week?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Of course! I'd be happy to help you understand this week's weather forecast. The outlook shows a mix of sunny and cloudy conditions.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"What about the temperature?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Temperatures will ranging from 12°C to 18°C throughout the week, with the warmest days being Wednesday and Thursday.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Should I bring an umbrella?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Yes, it would be advisable to carry an umbrella, particularly on Friday when there is a 60% chance of precipitation.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "result = fluency_evaluator(conversation=conversation)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ba89e73",
   "metadata": {},
   "source": [
    "#### Conversation as List[dict] - Agent with Tool Calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2658f610",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Conversation with tool calls - evaluates fluency of assistant responses\n",
    "conversation_with_tools = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Can you send me an email with the weather report for Seattle?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"I'll fetch the current weather information for Seattle and send you an email with the details.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"tool\",\n",
    "            \"content\": \"Temperature: 15°C, Condition: Partly cloudy, Wind: 8 mph NW\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"I have successfully sent you an email containing the weather report for Seattle. The current conditions are partly cloudy with a temperature of 15°C and light winds from the northwest.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "result = fluency_evaluator(conversation=conversation_with_tools)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff71d193",
   "metadata": {},
   "source": [
    "#### Conversation as List[dict] - Mixed Fluency Quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0d8bf46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Conversation with varying fluency levels across turns\n",
    "conversation_mixed = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"What's the status of my order?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Your order is being process currently and will be ship very soon to you.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"When will it arrive?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"The estimated delivery date for your order is March 15, 2025. You will receive tracking information via email once the package has been dispatched.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "result = fluency_evaluator(conversation=conversation_mixed)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
